There is growing interest in using Large Language Models as agents (LLM agents) for social simulations to inform policy, yet real-world adoption remains limited. This paper addresses the question: How can LLM agent simulations be made genuinely useful for policy? We report on a year-long iterative design engagement with a university emergency preparedness team. Across multiple design iterations, we iteratively developed a system of 13,000 LLM agents that simulate crowd movement and communication during a large-scale gathering under various emergency scenarios. These simulations informed actual policy implementation, shaping volunteer training, evacuation protocols, and infrastructure planning. Analyzing this process, we identify three design implications: start with verifiable scenarios and build trust gradually, use preliminary simulations to elicit tacit knowledge, and treat simulation and policy development as evolving together. These implications highlight actionable pathways to making LLM agent simulations that are genuinely useful for policy.

Crystal Cody is the Public Safety Technology Director for the City of Charlotte responsible for all technology related to Police, Fire, and the regional Radio Network. She has been in this role for six months and previously held the role of Computer Technology Solutions Manager for the Charlotte-Mecklenburg Police Department. In her twenty-year career with the City of Charlotte, she has been responsible for the selection, design, implementation, and management of all software applications used by the Police Department, and more recently, Public Safety. Her most notable accomplishments during this time are the implementation of CMPD's custom Records Management System, Computer Aided Dispatch system, the CRISS NC-LInX regional information sharing system, Predictive Analytics Business Intelligence Dashboards, and the Earl Intervention System along with a myriad of technology projects supporting the daily operations for Public Safety in Charlotte. Crystal worked closely with the staff at University of Chicago's Institute for Data Science and Public Policy in the development of the machine learning model used to identify officers at risk of adverse interactions with citizens and the CMPD has implemented use of the model through the development of an automated workflow for alerts and assessment dashboard.